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ABSTRACT 

In response to 1986 state legislation calling for 
improvements in college effectiveness and accountability, the 
California Community Colleges began various efforts to promote the 
evaluation of matriculation. One effort, the Matriculation Local 
Research Options Project, was formed in 1989 to develop models for 
local evaluation of matriculation and through June 1992 produced four 
major volumes of work. The first volume, produced in November 1989, 
described the goals of the project, indicating that the designs for 
evaluation would be simple, comprehensive, and operate within advised 
legal constraints. The work also provided nine research designs to be 
used as templates and adapted by local research offices. Volume II, 
published in February 1991, described findings bearing on the 
accuracy, utility, and fairness of assessment practices at system 
colleges, emphasizing the implications of minimum assessment 
standards published by the Chancellor's Office. Volume III was 
completed in February 1992 and described findings from a study of 
college use of the nine designs made available in Volume I. Data 
indicated that fewer reports from colleges using the designs were 
received than anticipated, due in part to changes in assessment 
practices stemming from the publication of assessment standards by 
the Chancellor's Office. Finally, Volume IV provided guidance to 
colleges in the following five areas of matriculation research 
identified as problematic or not yet addressed in previous volumes: 
(1) monitoring instructor grading practices; (2) multiple measures; 
(3) validation of course prerequisites; (A) follow-up activities; and 
(5) information tracking systems. (KP) 
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Matriculation in the California Community Colleges 

The Seymour-Campbell Matriculation Act (Assembly Bill 3) was passed by the 
state legislature in 1986. The bill represents one aspect of the reform measures 
introduced during the 1980's to improve the effectiveness, success, and 
accoimtability of the staters community colleges. 

Matriculation is a process that enhances access to the community colleges and 
promotes and sustains the efforts of college students to succeed in their academic 
endeavors through intervention and consistent support services provided by faculty 
and staff. The goals of matriculation are to ensure that all students complete their 
college courses, persist to the next academic term, and achieve their educational 
objectives. Since its inception, the matriculation process in the California 
community colleges has carried the banner for ensuring that state residents and 
citizens will be treated equitably with respect to access into the college system and 
educational opportunity. 

The Board of Governors of the California Community Colleges adopted 
regulations and guidelines which require the colleges to submit plans detailing how 
the process is implemented at each college. The matriculation process consists of 
seven components; admissions, orientation, assessment, counseling and 
advisement, student follow-up, coordination and training, and research and 
evaluation. Five of these components provide direct services to students, introduce 
and direct them to valuable facilities and individuals who may facilitate their 
leaning and academic well-being. Two of the components assist the colleges and 
districts in the development of training practices, polices and methodologies that 
involve the entire college community in the matriculation process. These 
components also help to develop institutional capabilities for the evaluation, 
compliance, coordination and analysis of matriculation, while fostering important 
linkages throughout the institution that aid in maintaining the clarity and objects of 
the process. 



Histor' of the Project 

In 1988, researchers in the field began to focus concern on plans for 
evaluation of matriculation, both at the local and statewide level. During the 1989 
Annual Research Conference sponsored by the Community College League of 
California and the Research and Planning Group for the California Community 
Colleges, Jon Kangas, Dean of Academic Standards for the San Jose /Evergreen 
Community College District, suggested a strategy for promoting local research and 
maximizing its benefits. He proposed that as researchers in the field, we should 
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develop a few simple models for local evaluation of matriculation, that we should 
publish the models and encourage their use wherever appropriate in the state, and 
that we should compile the results of each model in an aggregated report. 

The Chancellor's Office wanted to promote local research efforts in 
evaluating matriculation and was interested in providing technical assistance. In 
the spring of 1989, the Chancellor's Office called together a group of representatives 
from the Research and Planning Group and the Matriculation Regional Advisory 
Committee. The group was charged with executing the plan presented at Asilomar 
by Jon Kangas. This docimaent reviews the products of the group. 



Volume 1: MATRICULATION LOCAL RESEARCH OPTIONS PROJECT 

(November 1989) 

(This material is available from ERIC: ED 311 979/JC 890 485) 

Goals of the Project 

Simplicity : A paramount concern in the development of the first set of 
designs was that they be simple. We wanted straightforward studies that were 
simple to perform, and results that were easily imderstood by all audiences. We 
sought designs that could be executed even at colleges with very few resources for 
research. Nevertheless, to a degree, the strengths of this simplicity are balanced by 
methodological weaknesses. We acknowledge that the methodology of some of the 
studies could be improved through more elaborate design or analysis, but such 
changes would run counter to our goals for the project. 

Comprehensiveness : We have not developed a comprehensive or 
exhaustive evaluation of matriculation. Instead, we have developed a set of tools 
that are easy to use and fit many local needs. Our purpose is to promote local 
research and to aggregate it wherever sensible, but a complete evaluation of 
matriculation will need to go beyond these models. 

Colleges should not assume that completing a few or even all of the designs 
will constitute a comprehensive and exhaustive evaluation of matriculation. 
Additional research will be needed, including validation studies of local placement 
instruments and placement procedures, counts of students by exemption category, 
and other data as required by the Chancellor. 

Le gal constraints : The methodology of the designs was constrained by legal 
advice from the Chancellor*s Office. We were prohibited from presenting 
experimental designs that involved an untreated (or waiting-list) control group and 
assignment of subjects to treatment conditions. As a consequence, many of the 
designs rely on somewhat more complex, yet less powerful quasi-experimental 
approaches. 



The Designs 



Models : Each of the designs or studies is presented as a research template or 
model, and some of them may be adapted for use in your local district. Although 
you should not change basic features of the design without consulting with the 
Technical Assistant listed with the study, you may freely adapt the study to varying 
subject matter. For example. Study 6 might be adapted to study the relationship 
between reading level and political science performance. Also, the student 
satisfaction survey associated with Study 9 will need to be adapted for your local 
needs. 

Strategic suggestions: Since many of the designs have student demographic 
information (e.g., age, gender, ethnicity, educational goal, etc.) as variables, it would 
be advantageous to develop a single common data base. This data base could be 
utilized with the various studies as certain outcome information is compiled. One 
of the critical variables will be the identification of the "served" and "not-served" 
matriculants. Attachment 1 in Design 9 provides a survey form that could be used 
to gather the demographic data referred to in many of the designs. 

Technical assistance: We encourage you to contact the Technical Assistants 
for a iy of the designs you adopt. The Assistant is available to discuss the study and 
your plans, and to furnish technical advice on design, execution, data analysis and 
reporting. 

Statistical tests: Tests of statistical significance are not required in any of the 
first nine stiadies, although they are recommended if sample sizes fall below 100 per 
group, and because several of the designs do not meet the assumptions for tests of 
significance. We encourage you to focus your attention on the practical and 
educational significance of any differences you observe between groups. If the 
sample sizes exceed 100 per group, differences that are large enough to have practical 
importance will typically be statistically significant. 

Reporting : As part of accountability, we understand that the Chancellor's 
Office will be asking you to make available these final reports for the matriculation 
site review process. However, because our work group planned to prepare a 
monograph summarizing tiie findings across the colleges that perform each study 
reports were collected from the colleges. 



Volume III: MATRICULATION EVALUATION: MONOGRAPHS ON DESIGNS 
FROM THE LOCAL RESEARCH OPTIONS PROJECT 
(February 1992) 

(This material is available from ERIC: ED 345 787/JC 920 267) 



In November of 1989, nine research designs, aimed at assisting in evaluation 
of their matriculation of their matriculation activities locally, were disseminated to 
the California Commuruty Colleges. These designs were developed as part of the 
Local Research Options Project, sponsored by the Chancellor's Office, and focused on 
measuring the effects of matriculation on student performance, access and fairness 
issues, as well as student goal changes and satisfaction. Regional workshops were 
offered on the use of the designs and telephone consultation to the field was 
provided as necessary. Colleges were surveyed in October of 1990 regarding their use 
of the nine designs and their progress with the research studies. Colleges that had 
responded affirmatively received letters during the summer of 1991 soliciting their 
reports and /or data. The monographs contained in the third volume from the 
project summarize the findings generated from the use of the first nine designs. 

The purpose of the monographs is to aggregate the colleges' findings 
wherever feasible, to allow for a broad review of matriculation evaluation efforts 
and to provide pertinent feedback regarding the use of common research designs in 
future evaluation activities. Each monograph will delineate the purpose of the 
research design, the recommended methodology, the results for colleges submitting 
data and discuss the meaningfulness of the outcomes. When the methodology of 
the design was altered, it is noted within the monograph. In addition, difficulties 
that arose with the methodology will be addressed and possible modifications are 
proposed for future use of the design. 

There is a caveat regarding matriculation evaluation which is critical to keep 
in mind when reviewing the monographs. In July 1990, the Chancellor's Office 
published and distributed a document to the field. Standards, Policies and 
Procedures for the Evaluation of Assessment Instruments used in the California 
Community Colleges. Upon receipt of that document, colleges began to shift the 
focus of their research efforts from evaluating the overall impact of matriculation 
activities to validating their assessment instruments and practices. Limited 
resources were redirected to what appears to be a more vital need. Another set of 
designs related to the validity and fairness of assessment practices were developed 
and distributed to assist colleges with this effort. However, due to this shift in focus, 
there is much more activity occurring in assessment validation research than in 
evaluating the effectiveness of matriculation. 

Fewer reports were received for use in these monographs than anticipated. 
Although colleges probably are engaged primarily in research related to assessment 
validation, they may have been hesitant to submit their studies. A number of 
schools did not use the designs developed to assist them as they either already had 



research activities underway or had other evaluation models they were interested in 
utilizing. The Local Research Options Project committee members believed that 
there was much more research being conducted through the state on the 
effectiveness of matriculation than the limited niraiber of received reports reflected. 
The committee volunteered to survey the California community colleges on their 
matriculation evaluation research efforts undertaken since Fall 1989. The results of 
the survey £ire encouraging regarding the evaluation activities occurring in the 
colleges and are far greater than reflected in the document. 

Colleges have experienced a range of barriers that have frustrated their efforts 
in conducting research. These difficulties have been identified by colleges in their 
progress reports to the Chancellor's Office. These have to do primarily with lack of 
staff skilled in conducting research, time delays in modifying existing information- 
gathering procedures and computer support in compiling the necessary data. 
Although the colleges have been hampered in their desire to provide meaningful 
data that reflects on the effectiveness of their matriculation activities, the following 
monographs should elucidate initial findings that all colleges engaged in research 
activities can benefit from reviewing. 



Volume II: ASSESSMENT VALIDATION PROJECT LOCAL RESEARCH OPTIONS 

(February 1991) 

(This material is available from the Matriculation Unit of Chancellor's Office 
of the California Community Colleges) 



Purpose of the Volume 

The research designs presented in the third volume from the project were 
designed to yield a "Validation and Fairness Portfolio" - a set of findings bearing on 
the accuracy, utility and fairness of the assessment practices at the local college. 
Although the resulting information may not be definitive, it will be useful in any 
forum that reviews college assessment practices. Further, the portfolio is designed 
to respond to requirements published by the Chancellor's Office as minimum 
assessment standards to be met by local colleges or districts. 

Implications of the Minimum Assessment Standards 

In July 1990, the Chancellor's Office published Standards, Policies and 
Procedures for the Evaluation of Assessment Instruments Used in the California 
Community Colleges. Section III of that document represents a set of explicit criteria 
to be met by test publishes and another to be met by the colleges as test users. 

Scope of validation studies . Standard ILl.a. requires colleges to perform 
predictive validation and refers to Standard I.l.b. which states "Documentation is 
required to indicate that the test scores relate to performance in classes similar to 
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ones offered by the California conimunity colleges. The course content should bear 
a close logical relationship with the placement purpose of a test, not courses in 
general. " [italics otirs] 

This language is interpreted to indicate that predictive validation studies 
should focus on the relationship of placement recommendations to coturse 
performance in courses with content closely related to the test domain. That is 
math placement instruments should be validated against math course performance, 
'^vriting placement instruments should be validated against composition course 
performance, and so on. This standard does not require validation of placement 
tests as predictors of performance in courses in general. 

Assessments based on writing samples . Local colleges may wish to conduct 
predictive validity studies of assessments based on writing samples. As long as the 
writing assessments yield quantitative scores on a scale with equal intervals between 
numbers, those writing scores may be used like other scores in the designs presented 
here. In assessments based on writing samples, different types of writing prompts 
may result in differential performance by various student groups. 

Goals of the Volume 

Simplicity . In the 1989 Local Research Options publication, the desire was to 
present straight forward studies that were simple to perform and yielded results that 
were easily understood by all audiences. For the this volume, the goals were 
similar, but the issues are more complex, the methods are more technical, and the 
results will require more informed interpretation and qualification. For some of 
these designs, it is likely that support will be needed from the design's technical 
assistant or from the psychometric consultants working with the Chancellor's 
Office. In addition, computer data analysis is a practical requirement of several of 
the designs. 

Exhaustiveness . The project designs are not exhaustive of aL appropriate 
validation methods. A range of alternative procedures may be equally applicable to 
the key assessment issues. These designs merely represent examples of acceptable 
approaches for gathering evidence concerning the validity and fairness of 
assessment practices at local colleges. 

Practical constraints . Most colleges have assessment systems in place, and in 
many of these, placement practices have not been validated. Unfortunately, a 
number of technical considerations limit the usefulness of predictive validity 
evidence gathered on placement systems that are in use, although these limitations 
may be addressed by statistical corrections or by temporary modification or 
suspension of course screening practices. These constraints may mean that some 
colleges may be unable to gather conclusive predictive validity evidence. These 
colleges will need to supplement the predictive studies with research options that 
yield other types of validity evidence. 
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Evaluation as the Principle Purpose . The research designs presented in the 
volume are aimed at the collection of evidence about existing or planned 
assessment practices. They do not shov^ hovf to develop and implement new 
placement rules or decision systems. 

The Validation and Fairness Portfolio 

The Validation and Fairness Portfolio constitutes a set of findings bearing on 
the accuracy, utility and fairness of the assessment practices at the local college. 

It is sound practice to base validity estimates on a variety of appropriate 
evidence. Although the Minimum Standards focus on predictive validity evidence, 
the portfolio will be enhanced by the inclusion of additional evidence beyond that 
bearing on predictive validity. Other relevant information may include content 
validity evidence, student/faculty placement satisfaction data, and professional 
judgments. 



Volume IV: MATRICULATION EVALUATION: PHASE III 
LOCAL RESEARCH OPTIONS 
(June 1992) 

(This material is available from the Matriculation Unit of Chancellor's Office 
of the California Community Colleges) 



This is the third set of research designs (and fourth volume) developed 
through the Local Research Options Project. The purpose of this final document is 
to provide guidance to colleges in five areas of matriculation research that have 
been identified as problematic or have not yet been addressed in the previous 
booklets. 

The information provided in the volume is somewhat different than the 
presented in the previous documents as these are not so much research designs as 
they are recommended approaches to the five topics. As In the past, technical 
assistants are identified with each topic to provide additional consultation to the 
colleges. 

The first topic focuses on monitoring instructor grading practices. Although 
not the only nor error-free outcome measure, student final grades traditionally have 
been chosen as the criterion variable in research studies measuring student 
performance. Ease in collecting and compiling final grades is probably the primary 
reason they are chosen. When colleges are faced with limited resources in staffing, 
time and finances, final grades many times are the natural choice whether the 
research be related to ti\e effectiveness of their orientation programs, assessment 
instruments and practices, follow-up intervention strategies or validation of 
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prerequisites. The extent to which grading practices vary from instructor to 
instructor teaching the same course needs to be identified as to the impact it has on 
the statistical data generated for research purposes. 

Another critical area of concern has to do with how colleges can identify 
viable additional measures to incorporate into their placement practices. Although 
the term ''multiple measures'' has been bandied about for the past year, it has 
become necessary to provide guidance to colleges in the appropriate processes to 
imdertake to comply with the Matriculation Regulations. It is important to note 
that only assessment instruments need to be validated, not all methods or practices 
used in determining student placements. However, it is soimd educational practice 
to monitor the effectiveness of a placement rule which should contain multiple 
parts. 

The validation of course prerequisites has come to the forefront of 
matriculation issues as the regulations state that prerequisites and other limitations 
on enrollment need to be validated (with some exceptions-see Section 58106). 
Some schools have expressed concern about having prerequisites for their courses 
due to uncertainty on how to comply with the regulations. The design presented in 
this document is intended to provide colleges with clear direction on how to 
proceed in developing data bearing on the validity of their required prerequisites. 
This should allow colleges to have appropriate and realistic prerequisites for their 
courses and at the same time meet state requirements. 

The integration of follow-up activities has been one of the last areas addressed 
by many colleges in their implementation of matriculation. Although studies 
revolving around this component generally will follow traditional lines of research 
methodology, a variety of approaches may be utilized. The methodological format 
chosen for research will dictate how the follow-up activity is implemented and 
should be identified at the forefront of the endeavor. Seven different research 
formats are presented. 

Lastly, Matriculation and other requirements for monitoring performance 
over time will be with California Community Colleges for years to come. It is 
advantageous for schools to set up information tracking systems that can, with some 
ease, compile data that desaibe these outcomes. A guide is provided to assist 
colleges in collecting relevant information necessary in accountability efforts. 

It has been increasingly evident that research for matriculation activities is 
going to be a long-term effort. No school will be able to sit back and annoimce, 
"Now we are finished with our research". Assessment practices will require review 
every few years as cut scores are shifted or curriculum modified. As changes occur 
in colleges' demographics, the effectiveness of matriculation services provided to 
students will need to monitored. And hopefully. Follow-up activities will become 
an essential part of an institution's practices. Research needs to be incorporated as 
an integral part of an integral part of each college's mission. 
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